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Executive Summary 

In an effort to develop policy solutions to problems facing schools, there is often a 
tendency to view various initiatives as a panacea to address these challenges. Frequently, 
various stakeholders in the educational process propose a “silver bullet” to solve the 
problems of student achievement. Accordingly, proposals for reducing class size, 
increased funding, advanced training for teachers and a myriad of other options are pushed 
to the forefront of policy discussions. 

This paper seeks to answer the question, “What is the relationship between student 
achievement and four variables which are often cited as solutions?” Because supporters 
promote initiatives focusing on each of these variables, this paper seeks to identify which, 
if any, of the variables have the greatest impact on student achievement and whether there 
are any extenuating circumstances regarding that relationship. The selected variables are: 

1. Per pupil expenditure; 

2. Class size; 

3. Teacher advanced training; and 

4. Teacher experience 

These variables were selected because the data is routinely reported in publications such as 
the West Virginia Report Card and because they are often offered as potential “silver 
bullets”. Using comparative analysis, the effectiveness of each of these variables on 
student achievement was analyzed. 

In summary, findings show that: 

• Student achievement was not significantly different between the districts (i.e., 
counties) which had the highest per pupil expenditure and those districts that had the 
lowest per pupil expenditure. 

• Student achievement was not significantly different between the districts that had the 
smallest class sizes and those districts with the largest class sizes. 

• Student achievement was not consistently affected by teacher experience. 

• Student achievement was not consistently affected by teacher advanced tr aining . 

It was concluded that: 

• Within the parameters of the present data, none of these four variables demonstrates a 
strong enough direct relationship to student achievement in order to be recognized as 
a panacea or “silver bullet”. 

• Despite the attractiveness of a “simple solution” such as focusing on class size, per 
pupil expenditure, teacher training and experience, policy decisions need to be made 
on the basis of more complex issues such as how educational resources are utilized, 
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commitment to education and teaching, and instructional processes within the 
classroom regardless of its size. 



As much as we; would like simple solutions to complex problems, there are NO SIMPLE 
SOLUTIONS. 



INTRODUCTION : 



Intuitively it is appealing to believe that smaller classes should lead to better achievement 
by students. Teachers, with fewer charges, should be able to provide more individually 
determined instructional programs more suited to the needs of each child. With equal 
intuitive appeal, the idea that teachers with more experience and with advanced tr ainin g 
should produce higher achievement scores is also a long held belief system in educational 
history. Finally, the long held assumption that increased spending will produce higher test 
scores has provided a major policy making tenet across the country. It may well be that 
theses four intuitive concepts have become so entrenched as to lead us to inappropriate 
policy decisions. Yet there is a propensity to focus attention on single variables as a 
means of increasing student achievement. For example, the on-going initiative for the 
reduction of class size, the establishment of “five” year teacher education programs, 
increased pay for teachers to keep the experienced ones in the classroom, and “standards” 
based certification processes are but a few of the proposed solutions to increasing student 
achievement. In his now classic book. The Language of Education. Scheffler’s (1960) 
descriptions of slogans is particular relevant to the current policy making climate. Slogans 
are valuable to rally adherents of particular approaches or causes but slogans provide us 
with neither definitions of educational problems nor the data to make the best decisions. 
Human learning and the teaching process are complicated events. Yet, the public, 
stakeholders, and policy makers frequently offer one factor as the key to increasing 
student achievement. Add more money, add more teachers, consolidate, reduce class size, 
develop higher standards, the slogan list goes on and on. 

Concurrently, there is also another potentially critical tool that needs to continue moving 
beyond the slogan status. Namely the frequently call for “outcomes assessment”. There 
is a tendency to not look at outcome assessment as a means of determining the 
effectiveness of many of critical educational variables. This study begins to examine 
selected issues in terms of outcome assessment. Issues were selected from data available 
in the 1996-97,1997-98, and 1998-99 West Virginia Report Cards and the State 
Department of Education. 1 The selected topics are: 

1 . Per pupil expenditure; 

2. Classroom size; 

3. Average Level of Teacher Graduate Training; and 

4. Average Years of Teacher Experience 

The fundamental questions concern the relationship of each of these variables to the 
achievement of students. More specifically: 



1 Special thank you to Larry White, Coordinator, Data/Network Management and his staff at the West 
Virginia State Department of Education for supplying data not available in the Report Cards. 
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1. How does per pupil expenditure affect student achievemeiii? 

2. How does class size affect student achievement? ' 

3. Do teachers with more advanced graduate training affect higher achievement scores? 

4. Do students achieve better in school districts with more experienced teachers? - 

For the present study, the results on the Stanford Achievement Test (SAT) and Scores on 
the American College Test (ACT) served as indicators of achievement. To be sure, the 
sole purpose of schooling should not be measured just in terms of achievement test scores. 
Further, standardized achievement tests are restricted in their ability to assess the complex 
process of learning. 



REVIEW OF LITERATURE: 

V 

Intuition and or common sense do not always produce the best results. Credible research 
findings linking these four variables to student achievement is sometimes scarce and often 
inconsistent. 

Class Size 

Perhaps the most frequently offered proposal to increase student achievement is to 
decrease class size. The data generated by the STAR project in Tennessee usually 
provides the basis for these assertions. Recent reviews of research on the effectiveness of 
reducing class size have been presented by Ulig (1996), Wenglinksy (1999) and Hanusek 
(1999). 

CONSISTENT FINDINGS: 

1. Class Size has been reduced from over 30:1 in the 1950’s to less than 19:1 in 1999. 

2. Scores on national standardized tests have remained fairly constant despite this 
reduction in average class size. 

3. Data suggests that to be an effective variable, class size must be reduced to a 
maximum of 15 students per class. 

4. Most data indicate that disadvantaged children may benefit most from classes of less 
than 15 students. 

INCONSISTENT FINDINGS: 

1 . Data is absent as to the effectiveness of smaller classes where “small” is defined as 
more than 1 5, but less than 20. 

2. Analysis of Tennessee’s STAR Experiment data, which has provided much of the data 
for the smaller class mandate reveals that for children in kindergarten classes, class size 
of less than 15 clearly leads to increased achievement. However, the gains made in 
kindergarten did not show additional increases by remaining in smaller classes through 
the remainder of elementary school. This interpretation suggests that small classes, 
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that is less than 15, may be critical for pre-schooi kindergarten but the data does 
not support the continuing benefits of classes of 15 students or less. 

While Pate-Bain and her STAR colleagues (e.g. 1997) continue to offer indicators of 
maintaining the advantage of early experience in small classrooms, the data is less than 
overwhelming. Other studies by authors such as Hess (1978), McIntyre and Marion 
(1989) and a report from the federal government (1998) suggest “Overall findings do not 
support the cost associated with universal class size reduction and indicate that 
investments in other educational strategies may yield similar or greater achievement g ains 
(McIntyre and Marion, pg. 1). 



Teacher Experience and Advanced Training 

Reviews of the literature on the value of teacher experience and advanced tr aining in 
producing greater student achievement is remarkably thin. Thinness of outcome 
assessment data on this topic is not unusual, but the lack of research on these issues itself 
is startling given the major role these variables play in education policy decision making 
Gatthom (1997) reported a synthesis of the literature with the primary goal of providing 
East Carolina University with a data base for planning future graduate programs. He 
laments that information retrieval systems generated only four studies that provided more 
prescriptive than descriptive or self-report data. A meta-analysis by Greenwald, Hedges, 
and Laine (1996) concluded a positive link between teacher experience, graduate study 
and student achievement. Studies by Knoblock (1986) and Ferguson (1991) produced 
conflicting results. Glatthom continues in his policy analysis with a variety of thoughtful 
references to traditional educational theorists but cautions that any finding between 
increased student achievement and graduate study or teacher experience may be more an 
indicator of teacher’s commitment to education than an indicator of the benefits of 
graduate study or experience to students achievement. 

Goldhaber and Brewer (1996) report findings which suggest that teachers with 
certification and either bachelors’ or masters’ degrees in mathematics or science were 
associated with higher student performance scores in these topic areas. Bodenhausen 
(1988) found that advanced placement students tended to do better with teachers with 
more than ten years of experience. On the other hand, Keeler and McCall (1973) note 
“Analysis suggested that the relationship between experience and achievement was the 
ability of experienced teachers to become employed by schools with a student body of 
high achievers (pg. 1). 

Per Pupil Expenditures 

In 1959-60 the average per pupil expenditure for the United States was $375 per year. By 
1995-96 the level had increased to an average of $6,146. New Jersey had the highest 
average of $9,955 for 1995-96 and Utah had the lowest with an average of $3,867 for 
1995-96. 
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After the publication of Equality of Educational Opportunity in 1966, literally hundreds of 
studies have examined the relationship between per pupil expenditure and student 
achievement. These studies have produced a wealth of conflicting and controversial 
findings. In general however, there is agreement that the primary issue is the use made of 
the funds rather than the amount of funds. The more complicated issues of the u tilizatio n 
of resources rather the allocation of resources continues to present a research challenge. 



POLICY DECISION MAKING PROBLEMS: 

Educational researchers have tended to analyze much of their data in terms of correlation 
analysis. More specifically, a correlation indicates the degree of relationship between two 
variables. The higher the correlation, the stronger the relationship. This framework has 
long provided findings that are often conflicting, and frequently not directly applicable to 
policy decision making. This approach, can be called the “input-output regression model 
of research” and was designed to enable the prediction of outcomes or outputs based on 
the knowledge of inputs and their relationships to the outcomes. Unfortunately, to 
provide the information necessary for significant decision making, the relationship has to 
be very strong. When researchers report a correlation of .6 or .8 it indicates a fairly strong 
relationship. Even though a correlation may be statistically significant, there is often too 
much “noise” in the system, uncontrolled variations, to allow straight forward predictions. 
This is not to say that this approach is invalid or has not provided a huge pool of valuable 
information. It is to suggest that perhaps a change in perspective may provide information 
that is more specifically relevant to the needs of policy decision making. Relationship data 
tells us the degree of association between whole populations. If spends more money on its 
educational system is there a corresponding association with higher achievement scores? 
It may be more useful to ask comparing questions. This type of question is at the essence 
of outcome assessment decision makin g. 

“Do districts that spend more money have higher achievement test scores when compared 
to districts that spend less money?”. 

“Do districts that have more experienced teachers have higher achievement scores when 
compared to districts with less experienced teachers?”. 

“Do districts that have smaller classes have higher achievement scores when compared to 
districts that have larger classes?”. 

“Do districts with a higher percentage of advanced degree teachers score higher on 
achievement tests than districts with fewer advanced degree teachers?”. 

This approach to data analysis in educational research was presented in an article by 
Fortune (1993) with the specific purpose of assisting in policy decision making. The 
process also allows for analyzing when discrepancies make a difference. In traditional 
experimental psychology there is a concept called “just noticeable difference” or JND. 
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The JND marks the spot when a subject can tell that something different has happened. 
For example, with a hearing or vision examination, a person is asked to indicate when they 
cannot hear the tone or when they can see the letter in the fifth row. The point at which 
the person can make this differentiation is the JND — the point where there is enough 
discrepancy to tell a difference. 

In this approach to education research for policy decision making the question of when are 
we spending enough money to make a difference or what is the JND point for class size 
are vital concerns. Specifically, how small do classes have to be before size begins to 
affect achievement? At what point does teacher experience level or education level make 
a difference in student achievement? The approach suggested here allows us to generate 
specific outcomes assessment data that will provide policy makers with the relevant 
information to make significant policy decisions on the basis of research data rather than 
on the basis of the loudest or most frequently heard slogan. 



THE COMPARATIVE APPROACH: OUTCOMES ASSESSMENT 

Since we are now asking different kinds of questions — comparisons as opposed to 
associations — different methods and statistical techniques are required. The present study 
divided the 55 school districts into pools of “high” and “low” based on data for 1996-97 
and 1997-98 which was presented in the West Virginia Report Card. Information for the 
1998-99 school year was provided by the State Department of Education prior to the 
publication of the 1 998-99 Report Card. Data for these three years were used because of 
the change to the Stanford Achievement Test (SAT) which was made before the 1996 
school year. The American College Test (ACT) scores were used as an independent 
indicator of achievement. 

The ten districts with the highest per pupil expenditure for 1996-97 was called the high 
group and correspondingly, the ten districts with the lowest per pupil expenditure for 
1996-97 was called the low group. The districts were again sorted according to 1997-98 
data and 1998-99 data. 

The ten districts with the smallest class sizes for 1996-97 was called the small group and 
correspondingly, the ten districts with the largest class sizes for 1996-97 was called the 
large group. Again, the districts were sorted according to the 1997-98 data and 1998-99 
data. 

The ten districts with the most experienced educators for 1996-97 was called the high 
experience group and the ten districts with the least experienced educators for 1996-97 
was called the low experience group. Districts were again sorted into high or low 
according to the 1997-98 data and the 1998-99 data. 

The ten districts with the more advanced degree educators for 1996-97 was called the 
high degree group and the ten districts with the fewest advanced degree educators for 
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1996-97 was called the low degree group. Likewise, districts were again resorted 
according to 1997-98 data and the 1998-99 data. 

Different groupings were composed of different districts both within the same year and 
between years. On one variable, advanced degree, for example, districts that were in the 
low group for per pupil expenditure may place in the high advanced degree group or may 
not even be included in the sample districts. It should also be noted that group size ranged 
from nine to eleven depending on tied ranks. 

The next task was to compare the average achievement scores between each of the 
matching high and low groups. For example, the average, mean Stanford Achievement 
Test score for the 1996-97 High Per Pupil Expenditure group is 54.1 and the mean 
Stafford Achievement Test score for the 1996-97 Low Per Pupil Expenditure group is 
56.3. Obviously 56.3 is higher than 54.1; the question is whether or not there is a 
statistical difference once the “noise” in the system is taken into account. It may be that 
the low group has a higher mean because one district has an unusually high score or the 
high group may be lower because one district has an unusually low score. This is the type 
of “noise” which must be controlled before conclusions can be made. One standard 
statistical technique often used to permit such conclusions is called the t-test. Basically, 
the process allows for the researcher to say that one mean is different from another mean 
only if there is still a difference after we have taken the variance or “noise” into account. If 
the t-test value is large enough then it can be argued that the two mean scores are “realty 5 ’ 
different from each other. In our example, the t-test value is .92 which says there is no 
real difference between the two means. That is, 54.1 is not significantly different from 
56.3 once the variance or “noise” is taken into account. To have been statistically 
different, the t-test value in this case would have needed to be at least 2.13. Thus, we can 
conclude that the ten districts which had the highest per pupil expenditure for 1 996-97 did 
not have significantly higher achievement test scores when compared to the ten districts 
with the lowest per pupil expenditure for the 1996-97 school year. 

Care must be also given to the question of whether the two groups are significantly 
different from each other. That is, if the average amount spent by the high per pupil 
expenditure group is not different from the average amount spent by the low per pupil 
expenditure group, then we are not able to say that the two groups are really different. 
Again, the t-test technique comes to our assistance. For example, the mean expenditure 
for the 1996-97 high group is $6,840.70 per pupil and the mean expenditure for the 1996- 
97 low group is $5,653.90. The same question emerges. Are the two means different — is 
$6,840 different from $5,653 after all the noise has been accounted for? In this case, the 
t-test value was 11.49, much larger than needed to reach statistical significance. With 
virtually no doubt, the two groups are different in the amount of money each spends; 
therefore, we are clearly comparing the achievement of two different groups. 

This same process was applied to each of the four variables for each of the three years. 
The “high” groups were compared to the “low” groups and in all cases the compared two 
groups were different from each other. With this finding, it is appropriate to compare the 
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affects on achievement test scores. The results that follow compare average achievement 
test scores for the high and iow groups for each variable for each of the three years. 



FINDINGS: 

Tables I-IV present means and t-test values for both the Stanford Achievement Test and 
the ACT when districts were sorted by each of the four criteria. Again, t-test values were 
computed between the high and low groups and always indicated the appropriateness of 
treating the two groups as significantly different from each other. 

Per pupil expenditure for each of the three years is presented first. Mean scores by year 
andxomputed t-test values are presented in Table I. 



TABLE I 

PER PUPIL EXPENDITURE 



Comparison of achievement test scores and ACT scores for High versus 
Low per pupil expenditure groups 
average scores 



YEAR 


1996-97 


1997-98 


1998-99 


Mean score 


SAT* 


ACT* 


SAT 


ACT 


SAT 


ACT 


high group 


667.8 


19.71 


671.4 


20.59 


673.9 


19.9 


low group 


670.1 


20.57 


673.6 


19.66 


675.1 


20.5 




t = 1.19 


t = 2.29 


1! 

b 

to 


t = 2.90 


t = -0.52 


t = 1.13 




ND* 


D* 


ND 


D 


ND 


ND 



* Stanford Achievement Test 
*American College Test 
*ND equals No Difference 
*D equals Difference 

Findings indicate no differences between the high per pupil expenditure groups and the 
low expenditure groups on the Stanford Achievement Test Scores. Within the parameters 
of West Virginia’s expenditures, no affect on achievement was found with higher 
expenditures. The finding of differences between high and low groups on the ACT for 
years 1996-97 and 1997-98 were mixed. In 1996-97 the low expenditure group scored 
higher and in 1997-98 the high expenditure group scores higher. Since only selected 
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students take the ACT and given the restricted variance, these findings were not 
considered a significant finding. 



No Simple Solution: The school funding process is complicated. It depends on multiple 
funding sources, local economic levels, student special needs, salary levels, and a myriad 
of other factors. The present data clearly indicate, at least within the parameters of West 
Virginia’s per pupil expenditures, that the critical variables incorporate the uses made of 
the funds. Certainly, adequate funding is the necessary foundation for creating an effective 
learning environment. Creating an effective learning environment requires us to focus on 
complex processes which cannot be subsumed under the simplistic rubric of per pupil 
expenditure. 

Class size: 

Results comparing “large” class size to “small” class size are presented next. Rank 
ordering produced groups where large was defined as approximately 22.6 students and 
small was defined as approximately 17.8 students. While these differences are not large, t- 
tests computed between the groups by year indicated that the sizes were significantly 
different from each other. Comparative data is presented in Table II 

TABLE n 

Comparison of achievement test scores and ACT scores for 
large verses small class size groups. 



Average Scores 
YEAR 





1996-97 


1997-98 


1998-99 




MEAN SCORE 


SAT 


ACT 


SAT 


ACT 


SAT 


ACT 


large group 


672.6 


20.03 


675.4 


20.26 


677.0 


20.16 


small group 


670.6 


20.26 


673.8 


20.45 


675.5 


20.24 




t = 0.85 


t = 0.66 


t = 0.70 


t = 0.72 


t = 0.75 


t = 0.06 




ND 


ND 


ND 


ND 


ND 


ND 



No Simple Solution: Findings clearly suggest that smaller class size, at least within the 
parameters of 17 to 24 pupils per class, has no significant effect on achievement. 
Consistent with previous research, unless the class size is 15 or less, we should not expect 
any benefits to accrue to student achievement. As research also indicates, benefits of less 
than 15 are multidimensional and require careful consideration for policy decision making. 

TABLE HI 
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Years of Teacher experience 

Comparison of high experience group and low experience group 
on achievement test scores and ACT scores 



Average Scores 
YEAR 

1996-97 1997-98 1998-99 



Mean Score 


SAT 


ACT 


SAT 


ACT 


SAT 


ACT 


high group 


674.0 


20.52 


676.6 


20.46 


678.7 


20.49 


low group 


668.6 

D 


20.08 

ND 


679.9 

ND 


19.1 

ND 


674.9 

D 


20.14 

ND 



At first glance, findings indicate that teacher experience may have an affect on student 
achievement as measured by the Stanford Achievement Test. Hesitancy is present because 
of data presented in TABLE IV: Level of Teacher Education. A more complete 

discussion will follow presentation of this data. 

TABLE IV 

LEVEL OF TEACHER EDUCATION 
Comparison of achievement test scores and ACT scores 
for high versus low teacher education 



1996-97 


1997-98 


1998-99 




SAT 

MEAN SCORE 


ACT 


SAT ACT 


SAT 


ACT 


high group 668.3 


20.1 


672.1 20.1 


673.0 


20.2 


low group 672.5 


19.9 


676.0 20.3 


677.4 


20.1 


t = 2.03 


t = 0.66 


t = 2.19 t = 0.40 


t = 2.87 


OO 

o 

II 

+-» 


ND 


ND 


D ND 


D 


ND 



No Simple Solution: Findings indicate no differential affects on ACT scores. However, 
the low groups of graduate education consistently evidence higher achievement test scores 
as measured by the Stanford Achievement Test. This data presents a prime example of the 
“no simple solution” approach. When coupled with the teacher experience data, this 
finding suggests an seemingly inconsistent verdict. It would seem that experience and 
graduate training would be consistently related. Yet, our finding suggest the opposite. 
Clearly, if we were to accept either set of findings without knowledge of the other set, our 
policy making decisions would be based on inadequate information. Looking for an 
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answer at a deeper level may provide critical information. Hughes (1995; in her 
compelling study, Achieving Despite Adversity, consistently notes the importance of a 
experienced, well trained teaching cadre. But, even more consistently, and with greater 
emphasis, are the notations concerning: (a) commitment to education; (b) commitment 
to the school and the students; (c) educational leadership; and (d) low staff turnover. 

More experience does not necessarily include any of these characteristics nor does more 
graduate training. At the same time less experience and fewer graduate hours does not 
preclude these characteristics. The critical issues surround teacher commitment to 
education in general and the school and it’s students in particular. 



CONCLUSIONS: 

■Jr 

1. In general, higher per pupil expenditure, more years of experience, more 
graduate degrees, and smaller class sizes had little effect on the outcome 
assessments. Student achievement, as measured by the Stanford Achievement 
Test and the ACT remained relatively consistent across the data. 

2. The variations on each of the four variables while statistically different, were not 
sufficiently different to affect achievement as measured by the Stanford 
Achievement Test and the ACT. 

3. Policy decisions with regard to these four variables should focus on significant 
changes if any differences are to be noted. For example, unless class size is 
reduced to a maximum of 15 we should see no significant increases in 
achievement with minimal changes in average class size. 2 The same type of 
analysis applies to the remaining variables. 

4. Recall our concept of “just noticeable differences”. If these significant changes 
in funding, class size, etc. were to be introduced in schools, a new round of 
comparative investigations will be necessary to determine the ‘just noticeable 
differences”. 

5. Policy decision making can best be served at this time by focusing research 
attention to: 

A. Experimentation with variables on a trial basis which may lead 
the discovery of “just noticeable differences”; and 

B. Doing careful, in depth comparisons between districts, classrooms, 
teachers and a host of other variables that focus on outcomes 
assessments to isolate the determiners of increasing achievement. 

6. Comparative analysis can provide data that may be more useful to the decision 
making process than can regression data. 

7. Outcomes assessment using comparative analysis requires policy makers to 
approach decision making with demonstrated outcome data. Decisions based 
on the flexible nature of many regression analysis findings or educational slogans 
may lead to more accountability at all levels of the educational process. 

2 Size of 1 5 is suggested by National data. Even with a reduction to 1 5 or less, there is no consistent 

evidence of increased achievement. 
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Implications and Comments: 



Clearly care is needed not to over-generalize these findings. The findings were based on 
all the school districts in West Virginia. There is justification for transferability. The 
districts represent suburbs, urban areas and very rural areas. The per pupil expenditures 
for West Virginia are within the average for the nation. Teacher education programs are 
NCATE certified and like much of the nation, there is a clear “graying” of the teacher 
population. 

There should be no inferences that the author is not if favor of massive increases in school 
funding. Clearly the decaying state of the majority of physical plants and the minimal 
funding for educational programs when compared to such agencies as prisons, loudly 
speaks to the funding issue. Nearly forty years ago Barbara Tuchman noted: 

...we have, if anything a superabundance of leaders-hundreds 
of Pied Pipers, or would be Pied Pipers, running about, ready and 
anxious to lead the population. They are scurrying around, collecting 
consensus, gathering as wide an acceptance as possible (1967 pg.3) 

When coupled with the educational slogan dangers noted by Scheffler, all too often policy 
decisions are not based on accurate data or a proper educational research foundation. The 
ability to tread the minefield of slogans and stakeholders to carefully make policy that is 
clearly in the best interest of children and directly based on outcomes assessment is no 
simple feat. But then, there are NO SIMPLE SOLUTIONS. 
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